Hardware Accelerators Research Articles

TinyM 2 Net-V2: A Compact Low-power Software Hardware Architecture for M ulti m odal Deep Neural Networks

With the evaluation of Artificial Intelligence (AI), there has been a resurgence of interest in how to use AI algorithms on low-power embedded systems to broaden potential use cases of the Internet of Things (IoT). To mimic multimodal human perception, multimodal deep neural networks (M-DNN) have recently become very popular with the classification task due to their impressive performance for computer vision and audio processing tasks. This article presents TinyM 2 Net-V2 —a compact low-power software hardware architecture for m ulti m odal deep neural networks for resource-constrained tiny devices. To compress the models to implement on tiny devices, cyclicly sparsification and hybrid quantization (4-bits weights and 8-bits activations) methods are used. Although model compression techniques are an active research area, we are the first to demonstrate their efficacy for multimodal deep neural networks, using cyclicly sparsification and hybrid quantization of weights/activations. TinyM 2 Net-V2 shows that even a tiny multimodal deep neural network model can improve the classification accuracy more than that of any unimodal counterparts. Parameterized M-DNN model architecture was designed to be evaluated in two different case-studies: vehicle detection from multimodal images and audios and COVID-19 detection from multimodal audio recordings. The most compressed TinyM 2 Net-V2 achieves 92.5% COVID-19 detection accuracy (6.8% improvement from the unimodal full precision model) and 90.6% vehicle classification accuracy (7.7% improvement from the unimodal full precision model). A parameterized and flexible FPGA hardware accelerator was designed as well for TinyM 2 Net-V2 models. To the best of our knowledge, this is the first work accelerating multimodal deep neural network models on low-power Artix-7 FPGA hardware. We achieved energy efficiency of 9.04 GOP/s/W and 15.38 GOP/s/W for case-study 1 and case-study 2, respectively, which is comparable to the state-of-the-art results. Finally, we compared our tiny FPGA hardware implementation results with off-the-shelf resource-constrained devices and showed our implementation is faster and consumed less power compared to the off-the-shelf resource-constrained devices.

Lightweight Deep Learning for Resource-Constrained Environments: A Survey

Over the past decade, the dominance of deep learning has prevailed across various domains of artificial intelligence, including natural language processing, computer vision, and biomedical signal processing. While there have been remarkable improvements in model accuracy, deploying these models on lightweight devices, such as mobile phones and microcontrollers, is constrained by limited resources. In this survey, we provide comprehensive design guidance tailored for these devices, detailing the meticulous design of lightweight models, compression methods, and hardware acceleration strategies. The principal goal of this work is to explore methods and concepts for getting around hardware constraints without compromising the model’s accuracy. Additionally, we explore two notable paths for lightweight deep learning in the future: deployment techniques for TinyML and Large Language Models. Although these paths undoubtedly have potential, they also present significant challenges, encouraging research into unexplored areas.

Field Induced Off‐State Instability in InGaZnO Thin‐Film Transistor and its Impact on Synaptic Circuits

AbstractCharge storage synaptic circuits employing InGaZnO thin‐film transistors (IGZO TFTs) and capacitors are a promising candidate for on‐chip trainable neural network hardware accelerators. However, IGZO TFTs often exhibit bias instability. For synaptic memory applications, the programming transistors are predominantly exposed to asymmetric off‐state biases, and a unique field‐dependent on‐current reduction under off‐scenario is observed which may result in programming current variation. Further examination of the phenomenon is conducted with transmission line‐like method and degradation recovery tests, and current reduction can be attributed to contact resistance increase by charge trapping in the source and drain electrode and the channel region. The current decrease is subsequently formulated with a stretched exponential model with bias‐dependent parameters for quantitative circuit analysis under off‐state degradation. A neural network hardware acceleration simulator is utilized to assess the complicated impact the off‐state current degradation could instigate on on‐chip trainable IGZO TFT‐based synapse arrays. The simulation results generally demonstrate deteriorated training accuracy with aggravated off‐state instability, and the accuracy trend is elucidated from the perspective of weight symmetry point.

Survey on Redundancy Based-Fault tolerance methods for Processors and Hardware accelerators - Trends in Quantum Computing, Heterogeneous Systems and Reliability

Rapid progress in the CMOS technology for the past 25 years has increased the vulnerability of processors towards faults. Subsequently, focus of computer architects shifted towards designing fault-tolerance methods for processor architectures. Concurrently, chip designers encountered high order challenges for designing fault tolerant processor architectures. For processor cores, redundancy-based fault tolerance methods for fault detection at core level, micro-architectural level ,thread level , and software level are discussed. Similar applicable redundancy-based fault tolerance methods for cache memory, and hardware accelerators are presented in the article. Recent trends in fault tolerant quantum computing and quantum error correction are also discussed. The classification of state-of-the-art techniques is presented in the survey would help the researchers to organize their work on established lines.

HLS-IRT: Hardware Trojan Insertion through Modification of Intermediate Representation During High-Level Synthesis

Modern integrated circuit (IC) design incorporates the usage of proprietary computer-aided design (CAD) software and integration of third-party hardware intellectual property (IP) cores. Subsequently, the fabrication process for the design takes place in untrustworthy offshore foundries that raises concerns regarding security and reliability. Hardware Trojans (HTs) are difficult to detect malicious modifications to IC that constitute a major threat, which if undetected prior to deployment, can lead to catastrophic functional failures or the unauthorized leakage of confidential information. Apart from the risks posed by rogue human agents, recent studies have shown that high-level synthesis (HLS) CAD software can serve as a potent attack vector for inserting Hardware Trojans (HTs). In this paper, we introduce a novel automated attack vector, which we term “HLS-IRT”, by inserting HT in the register transfer logic (RTL) description of circuits generated during a HLS based IC design flow, by directly modifying the compiler-generated intermediate representation (IR) corresponding to the design. We demonstrate the attack using a design and implementation flow based on the open-source Bambu HLS software and Xilinx FPGA, on several hardware accelerators spanning different application domains. Our results show that the resulting HTs are surreptitious and effective, while incurring minimal design overhead. We also propose a novel detection scheme for HLS-IRT, since existing techniques are found to be inadequate to detect the proposed HTs.

Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis.

Graph neural networks (GNNs) are among the most powerful tools in deep learning. They routinely solve complex problems on unstructured networks, such as node classification, graph classification, or link prediction, with high accuracy. However, both inference and training of GNNs are complex, and they uniquely combine the features of irregular graph processing with dense and regular computations. This complexity makes it very challenging to execute GNNs efficiently on modern massively parallel architectures. To alleviate this, we first design a taxonomy of parallelism in GNNs, considering data and model parallelism, and different forms of pipelining. Then, we use this taxonomy to investigate the amount of parallelism in numerous GNN models, GNN-driven machine learning tasks, software frameworks, or hardware accelerators. We use the work-depth model, and we also assess communication volume and synchronization. We specifically focus on the sparsity/density of the associated tensors, in order to understand how to effectively apply techniques such as vectorization. We also formally analyze GNN pipelining, and we generalize the established Message-Passing class of GNN models to cover arbitrary pipeline depths, facilitating future optimizations. Finally, we investigate different forms of asynchronicity, navigating the path for future asynchronous parallel GNN pipelines. The outcomes of our analysis are synthesized in a set of insights that help to maximize GNN performance, and a comprehensive list of challenges and opportunities for further research into efficient GNN computations. Our work will help to advance the design of future GNNs.

High-speed hardware accelerator based on brightness improved by Light-DehazeNet

High-speed hardware accelerator based on brightness improved by Light-DehazeNet

QUIC on the fast lane: Extending performance evaluations on high-rate links

QUIC is a new protocol standardized in 2021 designed to improve on the widely used TCP/TLS stack. The main goal is to speed up web traffic via HTTP, but it is also used in other areas like tunneling. Based on UDP, it offers features like reliable in-order delivery, flow and congestion control, stream-based multiplexing, and always-on encryption using TLS 1.3.Unlike TCP, QUIC integrates these capabilities in user space, relying on kernel interaction solely for UDP. Operating in user space allows more flexibility but sacrifices some kernel-level efficiency and optimization that TCP benefits from. Various QUIC implementations exist, each distinct in programming language, architecture, and design.QUIC is already widely deployed on the Internet and has been evaluated, focussing on low latency, interoperability, and standard compliance. However, benchmarks on high-speed network links are still scarce.This paper presents an extension to the QUIC Interop Runner, a framework for testing the interoperability of QUIC implementations. Our contribution enables reproducible QUIC benchmarks on dedicated hardware and high-speed links. We provide results on 10G links, including multiple implementations, evaluate how OS features like buffer sizes and NIC offloading impact QUIC performance, and show which data rates can be achieved with QUIC compared to TCP. Moreover, we analyze different CPUs and CPU architectures influence reproducible and comparable performance measurements. Furthermore, our framework can be applied to evaluate the effects of future improvements to the protocol or the OS.Our results show that QUIC performance varies widely between client and server implementations from around 50Mbit/s to over 6000Mbit/s. We show that the OS generally sets the default buffer size too small. Based on our findings, the buffer size should be increased by at least an order of magnitude. Our profiling analysis identifies Packet I/O as the most expensive task for QUIC implementations. Furthermore, QUIC benefits less from AES NI hardware acceleration while both features improve the goodput of TCP to around 8000Mbit/s. The lack of support for NIC offloading from QUIC implementations results in missed opportunities for performance improvement.The assessment of CPUs from different vendors and generations revealed significant performance variations. We employed core pinning to examine if the performance of QUIC implementations is affected by the allocation to specific CPU cores. The results indicated an increased goodput of up to 20% when running on a specifically chosen core compared to a randomly assigned core. This outcome highlights the impact of CPU core selection on the performance of QUIC implementations but also for reproducible measurements.

SpikeExplorer: Hardware-Oriented Design Space Exploration for Spiking Neural Networks on FPGA

One of today’s main concerns is to bring artificial intelligence capabilities to embedded systems for edge applications. The hardware resources and power consumption required by state-of-the-art models are incompatible with the constrained environments observed in edge systems, such as IoT nodes and wearable devices. Spiking Neural Networks (SNNs) can represent a solution in this sense: inspired by neuroscience, they reach unparalleled power and resource efficiency when run on dedicated hardware accelerators. However, when designing such accelerators, the amount of choices that can be taken is huge. This paper presents SpikExplorer, a modular and flexible Python tool for hardware-oriented Automatic Design Space Exploration to automate the configuration of FPGA accelerators for SNNs. SpikExplorer enables hardware-centric multiobjective optimization, supporting target factors such as accuracy, area, latency, power, and various combinations during the exploration process. The tool searches the optimal network architecture, neuron model, and internal and training parameters leveraging Bayesian optimization, trying to reach the desired constraints imposed by the user. It allows for a straightforward network configuration, providing the full set of explored points for the user to pick the trade-off that best fits their needs. The potential of SpikExplorer is showcased using three benchmark datasets. It reaches 95.8% accuracy on the MNIST dataset, with a power consumption of 180 mW/image and a latency of 0.12 ms/image, making it a powerful tool for automatically optimizing SNNs.

Process-in-Memory realized by nonvolatile Task-Scheduling and Resource-Sharing XNOR-Net hardware Accelerator architectures

Due to the success of artificial intelligence systems in solving complex problems, these systems have been investigated in many applications. There is an increasing demand for these systems in many new applications. The XNOR-Net, a binary neural network, is a promising candidate for implementing these systems in devices with limited power and hardware resources. Efficient hardware implementation of XNOR-Nets is critical and has been widely investigated in recent research. In this paper, a resource-sharing gate and architecture and a task-scheduling gate and architecture have been proposed. To reduce power consumption, the proposed nonvolatile logic-in-memory gates are designed based on the magnetic tunnel junction (MTJ) and carbon nanotube field-effect transistor (CNTFET) devices. Using the proposed resource-sharing gate for hardware implementation of XNOR-Net minimizes the number of flip-flops, power consumption, PDP, and area by at least 75%, 75%, 60%, and 42%, respectively, as compared to its state-of-the-art counterparts. Also, using the proposed task-scheduling design to implement the XNOR-Net on 2, 4, and 8 datasets, the power consumption is reduced by at least 24%, 54%, and 65%, respectively, compared to its state-of-the-art counterparts .

AEKA: FPGA Implementation of Area-Efficient Karatsuba Accelerator for Ring-Binary-LWE-Based Lightweight PQC

Lightweight PQC-related research and development have gradually gained attention from the research community recently. Ring-Binary-Learning-with-Errors (RBLWE)-based encryption scheme (RBLWE-ENC), a promising lightweight PQC based on small parameter sets to fit related applications (but not in favor of deploying popular fast algorithms like number theoretic transform). To solve this problem, in this article, we present a novel implementation of hardware acceleration for RBLWE-ENC based on Karatsuba algorithm, particularly on the field-programmable gate array (FPGA) platform. In detail, we have proposed an area-efficient Karatsuba Accelerator (AEKA) for RBLWE-ENC, based on three layers of innovative efforts. First of all, we reformulate the signal processing sequence within the major arithmetic component of the KA-based polynomial multiplication for RBLWE-ENC to obtain a new algorithm. Then, we have designed the proposed algorithm into a new hardware accelerator with several novel algorithm-to-architecture mapping techniques. Finally, we have conducted thorough complexity analysis and comparison to demonstrate the efficiency of the proposed accelerator, e.g., it involves 62.5% higher throughput and 60.2% less area-delay product (ADP) than the state-of-the-art design for n =512 (Virtex-7 device, similar setup). The proposed AEKA design strategy is highly efficient on the FPGA devices, i.e., small resource usage with superior timing, which can be integrated with other necessary systems for lightweight-oriented high-performance applications (e.g., servers). The outcome of this work is also expected to generate impacts for lightweight PQC advancement.

Reconfigurable stochastic neurons based on strain engineered low barrier nanomagnets.

Stochastic neurons are efficient hardware accelerators for solving a large variety of combinatorial optimization problems. "Binary" stochastic neurons (BSN) are those whose states fluctuate randomly between two levels +1 and -1, with the probability of being in either level determined by an external bias. "Analog" stochastic neurons (ASNs), in contrast, can assume any state between the two levels randomly (hence "analog") and can perform analog signal processing. They may be leveraged for such tasks as temporal sequence learning, processing and prediction. Both BSNs and ASNs can be used to build efficient and scalable neural networks. Both can be implemented with low (potential energy) barrier nanomagnets (LBMs) whose random magnetization orientations encode the binary or analog state variables. The difference between them is that the potential energy barrier in a BSN LBM, albeit low, is much higher than that in an ASN LBM. As a result, a BSN LBM has a clear double well potential profile, which makes its magnetization orientation assume one of two orientations at any time, resulting in the binary behavior. ASN nanomagnets, on the other hand, hardly have any energy barrier at all and hence lack the double well feature. That makes their magnetizations fluctuate in an analog fashion. Hence, one can reconfigure an ASN to a BSN, and vice-versa, by simply raising and lowering the energy barrier. If the LBM is magnetostrictive, then this can be done with local (electrically generated) strain. Such a reconfiguration capability heralds a powerful field programmable architecture for a p-computer whereby hardware for very different functionalities such as combinatorial optimization and temporal sequence learning can be integrated in the same substrate in the same processing run. This is somewhat reminiscent of heterogeneous integration, except this is integration of functionalities or computational fabrics rather than components. The energy cost of reconfiguration is miniscule. There are also other applications of strain mediated barrier control that do not involve reconfiguring a BSN to an ASN or vice versa, e.g., adaptive annealing in energy minimization computing (Boltzmann or Ising machines), emulating memory hierarchy in a dynamically reconfigurable fashion, and control over belief uncertainty in analog stochastic neurons. Here, we present a study of strain engineered barrier control in unconventional computing.

Multilayer Neural Network Hardware Designs for Faster Convolutional Processing

The demand for efficient hardware implementations of convolutional neural networks (CNNs) has surged with the proliferation of deep learning applications in various domains such as computer vision, natural language processing, and autonomous systems. Convolutional layers, being the fundamental building blocks of CNNs, are computationally intensive, requiring optimized hardware architectures for real-time inference tasks. This paper presents novel hardware designs targeting faster convolutional processing within multilayer neural networks. We propose a multi-faceted approach that leverages parallelism, pipelining, and hardware acceleration techniques to enhance the efficiency of convolution operations. Our design optimally exploits the inherent data-level and model-level parallelism present in CNNs to achieve high throughput while minimizing latency.

Efficient Neural Ranking Using Forward Indexes and Lightweight Encoders

Dual-encoder-based dense retrieval models have become the standard in IR. They employ large Transformer-based language models, which are notoriously inefficient in terms of resources and latency. We propose Fast-Forward indexes—vector forward indexes which exploit the semantic matching capabilities of dual-encoder models for efficient and effective re-ranking. Our framework enables re-ranking at very high retrieval depths and combines the merits of both lexical and semantic matching via score interpolation. Furthermore, in order to mitigate the limitations of dual-encoders, we tackle two main challenges: Firstly, we improve computational efficiency by either pre-computing representations, avoiding unnecessary computations altogether, or reducing the complexity of encoders. This allows us to considerably improve ranking efficiency and latency. Secondly, we optimize the memory footprint and maintenance cost of indexes; we propose two complementary techniques to reduce the index size and show that, by dynamically dropping irrelevant document tokens, the index maintenance efficiency can be improved substantially. We perform an evaluation to show the effectiveness and efficiency of Fast-Forward indexes—our method has low latency and achieves competitive results without the need for hardware acceleration, such as GPUs.

Restricted Boltzmann Machines Implemented by Spin-Orbit Torque Magnetic Tunnel Junctions.

Artificial intelligence has surged forward with the advent of generative models, which rely heavily on stochastic computing architectures enhanced by true random number generators with adjustable sampling probabilities. In this study, we develop spin-orbit torque magnetic tunnel junctions (SOT-MTJs), investigating their sigmoid-style switching probability as a function of the driving voltage. This feature proves to be ideally suited for stochastic computing algorithms such as the restricted Boltzmann machines (RBM) prevalent in pretraining processes. We exploit SOT-MTJs as both stochastic samplers and network nodes for RBMs, enabling the implementation of RBM-based neural networks to achieve recognition tasks for both handwritten and spoken digits. Moreover, we further harness the weights derived from the preceding image and speech training processes to facilitate cross-modal learning from speech to image generation. Our results clearly demonstrate that these SOT-MTJs are promising candidates for the development of hardware accelerators tailored for Boltzmann neural networks and other stochastic computing architectures.

GraphSER: Distance-Aware Stream-Based Edge Repartition for Many-Core Systems

With the explosive growth of graph data, distributed graph processing becomes popular and many graph hardware accelerators use distributed frameworks. Graph partitioning is foundation in distributed graph processing. However, dynamic changes in graph make existing partitioning shifted from its optimized points and cause system performance degraded. Therefore, more efficient dynamic graph partition methods are needed. In this work, we propose GraphSER, a dynamic graph partition method for many-core systems. In order to improve the cross-node spatial locality and reduce the overhead of repartition, we propose a stream-based edge repartition, in which each computing node sequentially traverses its local edge list in parallel, then migrating edges based on distance and replica degree. GraphSER does not need costly searching and prioritizes nodes so it can avoid poor cross-node spatial locality. Our evaluation shows that compared to state-of-the-art edge repartition software methods, GraphSER has an average speedup 1.52x, with the maximum up to 2x. Compared to the previous many-core hardware repartition method, GraphSER performance has an average of 40% improvement, with the maximum to 117%.

Low-cost and efficient prediction hardware for tabular data using tiny classifier circuits

AbstractA typical machine learning development cycle maximizes performance during model training and then minimizes the memory and area footprint of the trained model for deployment on processing cores, graphics processing units, microcontrollers or custom hardware accelerators. However, this becomes increasingly difficult as machine learning models grow larger and more complex. Here we report a methodology for automatically generating predictor circuits for the classification of tabular data. The approach offers comparable prediction performance to conventional machine learning techniques as substantially fewer hardware resources and power are used. We use an evolutionary algorithm to search over the space of logic gates and automatically generate a classifier circuit with maximized training prediction accuracy, which consists of no more than 300 logic gates. When simulated as a silicon chip, our tiny classifiers use 8–18 times less area and 4–8 times less power than the best-performing machine learning baseline. When implemented as a low-cost chip on a flexible substrate, they occupy 10–75 times less area, consume 13–75 times less power and have 6 times better yield than the most hardware-efficient ML baseline.

Enhancing Regular Expression Processing through Field-Programmable Gate Array-Based Multi-Character Non-Deterministic Finite Automata

This work investigates the advantages of FPGA-based Multi-Character Non-Deterministic Finite Automata (MC-NFA) for enhancing regular expression processing over traditional software-based methods. By integrating Field-Programmable Gate Arrays (FPGAs) within a data processing framework, our study showcases significant improvements in processing efficiency, accuracy, and resource utilization for complex pattern matching tasks. We present a novel approach that not only accelerates database and network security applications, but also contributes to the evolving landscape of computational efficiency and hardware acceleration. The findings illustrate that FPGA’s coherent access to main memory and the efficient use of resources lead to considerable gains in processing times and throughput for handling regular expressions, unaffected by expression complexity and driven primarily by dataset size and match location. Our research further introduces a phase shift compensation technique that elevates match accuracy to optimal levels, highlighting FPGA’s potential for real-time, accurate data processing. The study confirms that the benefits of using FPGA for these tasks do not linearly correlate with an increase in resource consumption, underscoring the technology’s efficiency. This paper not only solidifies the case for adopting FPGA technology in complex data processing tasks, but also lays the groundwork for future explorations into optimizing hardware accelerators for broader applications.

Memristor-based hardware accelerators for artificial intelligence

Memristor-based hardware accelerators for artificial intelligence

FPGA-based hardware accelerator for SIC in uplink NOMA networks

FPGA-based hardware accelerator for SIC in uplink NOMA networks